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CLAIMS: 



1. A system (100) for integrative analysis of intrinsic (10) and extrinsic (11) 
audio-visual data, the system comprising: 

an intrinsic content analyser, the intrinsic content analyser being 
communicatively connected to an audio- visual source, the intrinsic content analyser being 
5 adapted to search the audio-visual source for intrinsic data and being adapted to extract 
intrinsic data using an extraction algorithm, 

an extrinsic content analyser, the extrinsic content analyser being 
communicatively connected to an extrinsic information source, the extrinsic content analyser 
being adapted to search the extrinsic information source and being adapted to retrieve 
10 extrinsic data using a retrieval algorithm, 

wherein the intrinsic data and the extrinsic data are correlated, thereby 
providing a multisource data structure. 

2. A system according to claim 1, wherein the retrieval of the extrinsic data is 
15 based on the extracted intrinsic data. 

3. A system according to claim 1, wherein the extraction and/or retrieval 
algorithm(s) is/are provided by a module. 

20 4. A system according to claim 1, wherein a query is provided by a user, the 

query being provided to the extraction algorithm and wherein the intrinsic data is extracted in 
accordance with the query. 

5. A system according to claim 1, wherein a query is provided by a user, the 

25 query being provided to the retrieval algorithm and wherein the extrinsic data is retrieved in 
accordance with the query. 



6. A system according to claim 1, wherein a feature reflected in the intrinsic and 

extrinsic data include textual, audio and/or visual features. 
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7. A system according to claim 1, wherein the audio-visual source is a film (101) 

and wherein the extracted data include textual (104), audio and/or visual features (105, 106). 

5 8. A system according to claim 1, wherein the extrinsic information source is 

connected to and may be accessed via the Internet (103). 

9. A system according to claim 1, wherein the extrinsic information source is a 
film screenplay (102). 

10 

10. A system according to claim 9, wherein the extrinsic content analyser include 
knowledge about screenplay grammar, and wherein the extrinsic data is retrieved based on 
information extracted from the screenplay by use of the screenplay grammar. 

15 11. A system according to any of the claims 9 or 1 0 wherein the identification (5) 

of persons in a film is obtained by means of the screenplay. 

12. A system according to any of the claims 9 or 10 wherein a feature in a film is 
analysed based on information included in the screenplay. 

20 

13. A system according to claim 1, wherein the correlation of the intrinsic and 
extrinsic data is time correlation (121), thereby providing a multisource data structure where 
a feature reflected in the intrinsic data is time correlated to a feature reflected in the extrinsic 
data. 

25 

14. A system according to claim 13, wherein the time correlation is obtained by an 
alignment of a dialogue (120) in the screenplay to the spoken text (104) in the film and 
thereby providing a timestamped transcript (121) of the film. 

30 15. A system according to claim 14, wherein a speaker identification in the film is 

obtained from the timestamped transcript. 

16. A system according to claim 9, wherein the screenplay is compared with the 

spoken text in the film by means of a self-similarity matrix (30). 
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17. A system according to claim 1, wherein a high-level information structure (5- 

9) is generated in accordance with the multisource data structure. 

5 18. A system according to claim 17, wherein the high-level information structure 

may be stored on a storage medium. 

19. A system according to claim 17, wherein an updated high-level information 

structure is generated, the updated high-level information structure being an already existing 
10 high-level information structure which is updated in accordance with the multisource data 
structure. 



20. A system according to claim 1, wherein the retrieval algorithm is a dynamic 
retrieval algorithm adapted to dynamically update itself by including additional 

15 functionalities in accordance with retrieved extrinsic data. 

21. A system according to claim 20, wherein the additional functionalities is 
obtained by training the retrieval algorithm on a set of features from intrinsic data using : 
labels obtained from the extrinsic data. 

20 

22. A system according to claim 9 and 21, wherein the training is performed using 
at least one screenplay. 

23. A system according to 1, wherein an automatic ground truth identification in a 
25 film is obtained based on the multisource data structure for use in benchmarking algorithms 

on audio-visual content. 



24. A system according to 1, wherein an automatic scene content understanding in 

a film is obtained based on the textual description in the screenplay and the audio-visual 
30 features from the film content 



25. A system according to 1 , wherein an automatic labelling in a film is obtained 

based on the multisource data structure. 
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26. A method for integrative analysis of intrinsic and extrinsic audio- visual 

information, the method comprising the steps of: 

searching an audio-visual source for intrinsic data and extracting intrinsic data 
using an extraction algorithm, 
5 searching an extrinsic information source and retrieving extrinsic data using a 

retrieval algorithm, 

correlating the intrinsic data and extrinsic data, thereby providing a 
multisource data structure. 

10 27. A method according to claim 26 further comprising the step of generating a 

high-level information structure in accordance with the multisource data structure. 

28. A method according to claim 26, wherein the extrinsic content analyser 
include knowledge about screenplay grammar, and wherein the extrinsic data is retrieved 

1 5 using information extracted from the screenplay by use of the screenplay grammar. 

29. A method according to claim 26; wherein the retrieval algorithm is updated by 
training the algorithm on a set of extrinsic data. 

20 30. Computer programme product enabling a computer to be programmed to 

perform the method according to claim 26. 

3 1 . Storage medium canying the computer programme product according to claim 
30. 

25 

32. Programmed computer enabled to perform the method according to claim 26. 



